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OVERVIEW  OF  THE  ‘CRAMPS’  MULTIPROCESSOR  OPERATING  SYSTEM 


Peter  Mansbach 

National  Institute  of  Standards  and  Technology 
Bldg.  220/Rm.  B-124 
Gaithersburg,  MD  20899 


ABSTRACT 

CRAMPS  is  an  operating  system  designed  for  use  by  a number  of  independent  functicxially-divided 
processors  which  communicate  via  common  (shared)  memcay.  It  achieves  high  speed,  simplicity, 
and  ready  adaptability  to  users’  individual  needs  by  being  primarily  a single-tastog  system,  and 
adding  processor  boards  when  concurrent  t^jeration  of  processes  is  required.  It  thus  trades  software 
cost  for  known  hardware  cost 

Three  mechanisms  of  asynchrtxious  data  conununication  between  processes  are  provided.  These 
include  a set  (tf  f/nix-compatible  calls,  so  a program  can  be  debugged  in  a Unix  environmoit  and 
simply  re-linked  to  run  on  a targ<^  board.  CRAMPS  allows  individual  ptocesstvs  to  be  stopped  and 
restarted,  without  interrupting  the  other  processors,  and  allows  single  [uocesscus  to  be  run  in  isola- 
tion. The  operating  system  includes  a fast  downloeder,  a monitcv  (in  PROM),  and  an  array  of 
debugging  tools.  CRAMPS  inovides  an  extremely  fast  system  for  single-tasking  multiple-processor 
applications,  and  some  multi-tasking  capability  as  well.  It  is  currently  running  on  Motorola  680x0 
microprocessOTS,  and  has  also  run  on  Intel  8086's. 


keywords:  asynclmmoiis  communication;  communicatioas  protocol;  functionally-divided  inocesses; 
CRAMPS;  multiprocessing;  muMptocessor;  operating  syston;  real-time;  robot  vision;  vision. 


OVERVIEW  OF  THE  ‘CRAMPS’  MULTIPROCESSOR  OPERATING  SYSTEM 


I.  Introduction 

GRAMPS“the  General  Real-time  Asynchronous  Multi-Processor  System— is  an  operating  system 
designed  for  use  by  a number  of  independent  functionally-divided  processors  which  communicate  with 
each  other  via  common  (shared)  memory.  CRAMPS  is  currently  running  on  Motorola  680x0  microproces- 
sors, several  of  which  communicate  over  a VME  backplane.^  An  earlier  version  has  also  run  on  Intel  8086 
processors,  with  a Multibus  backplane. 

CRAMPS  is  very  fast  and  very  simple.  It  can  be  easily  modified  and  tailwed  to  users’  individual 
needs.  It  achieves  these  qualities  by  being  primarily  a single-tasking  system,  thus  avoiding  the  complexity 
of  "time-sharing"  or  multitasking  systems.  When  concurrent  operation  of  processes  is  required,  additional 
processor  boards  are  easily  added;  CRAMPS  handles  the  communications.  Thus,  additional  hardware  (if 
necessary)  is  substituted  for  more  complicated,  expensive,  and  errOT-prone  software. 

CRAMPS  supports  both  synchronous  and  asynchronous  transfer  of  messages  or  other  data  between 
processors.  Further,  three  mechanisms  of  asynchronous  data  transfer  are  provided,  depending  on  the  needs 
and  preferences  of  the  user.  A subset  of  these  communications  calls,  including  open,  close,  read,  write, 
prinrf,  getchar,  and  putchar,  use  the  HO  calling  formats.  Thus  a program,  which  is  written  and  com- 
piled on  a Unix  host,  may  be  linked  to  the  standard  Unix  library  and  debugged  on  the  host.  The  same  pro- 
gram may  be  re-linked  with  the  CRAMPS  library  and  downloaded  to  the  target.  On  the  host  the  "interpro- 
cesstH*"  communication  takes  place  through  files  rather  than  common  memory. 

CRAMPS  allows  individual  processors  to  be  stopped  and  restarted,  without  intmnpting  the  other 
processors,  and  allows  single  processors  to  be  run  in  isolation. 

CRAMPS  was  originally  designed  to  have  each  processor  run  a single  task.  In  this  mode  the  operat- 
ing system  executes  only  when  requested  by  the  usct  process.  This  permits  the  individual  processes  to 
thieve  their  full  speed  potential.  Also,  the  absence  of  multi-tasking  allows  great  simplicity  in  the  operat- 
ing system,  compared  to  "time-sharing"  systems,  and  in  particular  in  the  writing  of  input/ouq>ut  device 
drivers,  and  in  the  use  of  interrupts. 

Functions  have  since  been  added  to  allow  simple  multi-tasking  (^ration.  System  calls  to  activate 
and  de-activate  tasks  are  provided,  and  the  on-board  timer  is  programmed  to  interrupt  the  rurming  task  at  a 
time  specified  by  the  schedule.  Thus  the  user  can  now  perform  efficient  context-switching,  subject  to  a 
few  constraints.  Since  simple  priority  scheduling  is  oftra  insufficient  to  satisfy  the  needs  of  real-time 
an}lications,  the  user  himself  can  specify  and  {HOgram  the  exact  scheduling  algorithm  required. 

Numerous  checks  and  tools  are  provided  fcv  aid  in  debugging.  These  may  be  turned  off  to  permit 
still  faster  execution  and  smallo'  code. 

Most  of  the  CRAMPS  code  is  written  in  C.  Some  of  the  basic  routines-the  message-passing  primi- 
tives, terminal  HO,  block  moves,  etc.-are  in  assembly  language,  as  are  portions  of  the  PROM. 

1.  Background 

The  CRAMPS  operating  system  was  develq)ed  by  the  authcR’  at  the  National  Bureau  of  Standards. 

This  woik  was  done  in  suj^xirt  ot  a robot  vision  system  which  was  initially  constructed  with  five  Intel 
8086  microprocessOTS  in  a Multibus  backplane  [1,2,3].  Memory  was  {Hovided  that  was  accessible  to  all 
the  processors  via  the  Multibus. 

It  was  desired  to  run  the  diffnent  stages  of  vision  processing  concurrently  on  separate  processors, 
and  to  pass  data  from  one  stage  to  anotho'  asynchronously.  Data  transfo’  would  take  place  via  common 

1 Comm»rci«l  produca  ar*  klantiflad  in  ontar  iduuMiH'  id  dMOrib*  thD  equipment.  In  no  case  does  such  identification  imply  recommendation 
Of  endorsement  by  the  National  Institute  of  Siandarda  «td  Technolo»  nor  does  it  Imply  that  Ihe  equipmont  identified  is  necessarily  tiie  best  available 
for  the  purpose. 
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memory,  by  having  the  processor  that  generated  the  information  write  it  into  common  memory,  and  hav- 
ing each  processor  that  needed  that  information  read  it  when  required.  Mutual  exclusion,  to  prevent  one 
stage  from  reading  data  while  another  was  still  writing,  and  vice  versa,  was  to  be  provided  by  the  operat- 
ing system. 

Commercial  operating  systems  available  at  the  time  did  not  support  multiprocessor  data  passing. 
(We  use  the  term  multiprocessor  to  refer  to  systems  with  two  or  more  processors,  as  distinct  from  multi- 
tasking, where  several  tasks  may  execute  on  the  same  CPU.)  Since  data-passing  between  processors  was  a 
central  requirement  for  our  vision  system,  it  seemed  appropriate  to  write  a simple  operating  system  of  our 
own.  In  addition,  for  a research  project  with  unanticipated  requirements  there  is  a substantial  advantage  to 
writing  one’s  own  operating  system,  and  thereby  having  both  the  source  code  and  the  expertise  to  modify 
it  readily  available.  The  system  is  then  easily  a^pted  to  the  needs  of  the  project. 

We  adt^ted  the  following  design  requirements  for  the  CRAMPS  operating  system: 

Support  for  multiple  processors 

Efficient  data  passing  via  shared  memory,  often  large  blocks  of  data 
Support  for  txxh  synchronous  and  asynchronous  communication 
Unix-like  I/O  calls 

Ability  to  restart  one  pocess  while  others  continue  uninterrupted 
Ability  to  run  single  processes  without  others  present 
Specific  identifying  messages  for  all  errors  in  system  code 
Dynamic  common  memory  allocation 
Task-switching  capability 


2.1.  Other  Multiprocessor  Systems 

Since  that  time  a number  of  other  systems  have  become  available  with,  or  have  added,  multiproces- 
sor capabilities.  These  include  GEM  [4],  Harmony  [5],  Meglos  [6],  MTOS  [7],  POPEYE  [8],  pSOS  [9], 
VRTX  [10],  and  VxWorks  [11]. 

GEM  (devel(^)ed  at  Ohio  State)  envisions  a process  architecture  quite  diffo^nt  from  ours,  with 
processes  being  dynamically  assigned  to  pocessors,  and  containing  "micro-pocesses’’  which  can  be  turned 
on  and  ofr  by  oth^  micro-processes.  Harmony  (National  Research  Council,  Canada)  is  in  some  ways 
similar  to  CRAMPS,  but  has  more  highly  developed  multi-tasking  facilities.  However,  the  Harmony  system 
blocks  when  sending  or  receiving  messages,  which  may  lose  a fair  amount  of  potradal  processing  time  in 
an  application  like  ours  that  is  not  multi-ta^dng.  Meglos  (developed  at  Bell  Laboratories)  does  not  use 
shar^  memory,  but  instead  bases  its  communications  on  the  elaborate  and  very  fast  SINET,  also 
developed  by  ^11  Labs. 

MTOS  (Industrial  Programming  Inc.),  pSOS  (Software  Components  Group),  and  VRTX  (Hunter  & 
Ready)  are  all  systems  initially  designed  to  be  multi-tasking  on  a single  processor,  which  have  now  been 
extended  to  allow  multiprocessor  operations  as  weU.  MTOS  additionally  offers  dynamic  load-balancing 
among  the  processcus.  POPEYE  (Camegie-Mellon)  is  a complete  vision  system,  including  a 68000-hased 
operating  system  widi  many  features  similar  to  CRAMPS.  However,  there  is  no  mention  in  the  cited  refer- 
ence of  communication  firom  one  processor  board  to  another.  And,  finally,  VxWorks  (Wind  River  Sys- 
tems) runs  using  VRTX  as  the  operating  system  for  each  board,  and  uses  the  Unix  socket  formalism  to 
implement  interprocessor  communications. 

3.  Major  Components  of  the  Operating  System 

The  full  CRAMPS  operating  system  consists  of  five  physically  distinct  parts:  kernel,  downloader, 
PROM,  SYS  (system  process),  and  debugging  tools. 

3.1.  Kernel 

The  major  part  of  the  system  code  is  the  kernel,  which  is  a collection  of  subroutines  resident  in  the 
CRAMPS  run-time  library.  These  subroutines  are  included  by  the  linker,  and  thus  become  part  of  each 


2 Unix  u a trademaik  of  ATdtT/Bell  Labontories. 
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processor’s  program  code.  The  kernel  includes  the  functions  dealing  with  common  memory  communica- 
tion with  other  processes,  terminal  I/O,  dynamic  memory  allocation,  process  initialization,  etc.  These  are 
discussed  further  below. 

Note  that  the  C language  promises  that  initialized  variables  start  out  with  the  values  specified  by  the 
programmer,  but  these  values  may  change  during  program  execution.  Thus,  to  allow  a program  to  be  res- 
tarted, the  GRAMPS  kernel  saves  these  initial  values  the  first  time  the  program  is  executed  after  being 
downloaded,  and  subsequently  restores  them  each  time  the  process  is  restarted.  It  is  not  necessary  to  re- 
download  the  program  to  restart  it. 

3.2.  Downloader 

The  downloader  is  resident  on  the  host  computer,  and  sends  fully  compiled  and  linked  programs  to 
the  target  processor  boards  for  execution  there.  These  programs  are  sent  first  to  an  interface  board  on  the 
target  computer.  From  there  the  destination  processor  copies  them  to  its  on-board  memory  under  control  of 
its  PROM.  Clearly  the  choice  of  host  computer  is  not  important  to  the  rest  of  the  GRAMPS  system,  and 
can  be  changed  to  suit  the  application. 

The  downloader  performs  some  alto^ons  to  the  object  code  before  sending  it  to  the  target  board. 
In  particular,  it  finds  the  symbol  table  pM-oduced  by  the  compilCT,  compresses  it  and  appends  it  to  the  code, 
for  use  by  the  GRAMPS  tools.  It  also  determines  the  highest  program  address,  and  inserts  that,  the  stan- 
ing  address,  and  the  symbol  table  address  into  a table  used  by  the  startup  subroutines  and  the  PROM.  The 
code  itself  is  sent  in  a compacted  format  less  than  half  as  long  as  the  common  Motorola  5-format,  and 
thus  taking  less  than  half  the  time  to  download. 

The  download  p)rogram  is  indep)endent  of  the  device  hardware,  except  that  it  calls  different  device 
drivers  for  different  hardware.  It  has  been  run  using  standard  serial  RS-232  non-login  lines,  and  alterna- 
tively using  16-bit  parallel  HO  ports.  This  last  is  p)articularly  fiasL  The  download  pnogram  could  also  use  a 
network  link,  such  as  an  ethernet  line,  or  could  load  over  the  VM£bus  if  the  host  and  target  p}rocessors 
share  the  same  backplane. 

33.  PROM 

The  PROM  is  semi-p)ennanent  code  'Iximed  in"  to  read-only  memory,  one  copy  per  p)rocessOT 
board.  It  includes  a wakeip  sequence  fw  the  board,  the  "receiving  (townloader"  for  receiving  data  from  (or 
sending  it  to)  the  host  or  development  computer,  monitor  functions  for  directly  examining  registers  and 
memory,  and  other  basic  debugging  aids.  An  onboard  timer  chip,  if  one  exists,  can  be  selected  to 
automatically  time  sections  of  code.  A help  conunand  reviews  the  available  monitor  commands  and  for- 
mats. 

In  addition  to  the  usual  monitor  commands  (display,  substitute,  go,  go  to  breakpoint,  single  step, 
move,  fill,  find,  scan,  compare,  reset,  etc.),  we  have  added  a go  next  n command,  that  goes  through  the 
next  break^int  (n-1)  times  before  finally  stopping  on  the  nth  time. 

Another  nice  feature,  in  die  PROM's  wakeup  piiocedure,  is  diat  it  does  not  automatically  wait  fOT 
input  from  the  tenninal.  Rather  it  cyclically  pxills  the  terminal,  the  parallel  pwrt,  the  serial  link  to  the  host, 
and  a pae-defined  PROM  message  buffer.  Thus  a user  on  the  host  computer  may,  by  activating  the  host’s 
download  pirogram,  cause  the  interface  board’s  pnocessor  to  enter  it’s  "receiving  downloader"  pirogram, 
either  for  serial  or  piarallel  transmission.  It  is  not  necessary  to  first  ^ter  a command  firom  the  interface 
board’s  terminal;  in  fact,  it  is  not  necessary  to  have  a tenninal  there  at  all.  When  the  pnogram  has  been 
transfored  to  the  interface  board,  the  host’s  download  pnogram  will  then  instruct  the  interface  board  to 
store  a command  in  the  target  board’s  PROM  message  buffer.  The  target  board,  polling  on  this  buffer,  will 
receive  the  command,  enter  a subroutine  in  its  PROM  that  copies  the  code  from  the  interface  board’s  com- 
mon memory  into  the  target  board’s  local  memory,  and  (if  so  commanded)  start  executing  the  program. 
This  is  entirely  automatic,  and  in  re^xmse  to  a single  download  coipmand  typed  on  the  host 

3.4.  SYS 

The  system  process,  SYS,  consists  pnimarily  of  initialization  functions,  including  setting  up  the  mes- 
sage buffer  directory  in  common  memory,  writing  out  a table  of  system-wide  parameters  (such  as  the 
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camera  calibration  parameters  used  in  the  vision  system),  and  zeroing  buffer  areas  for  easier  debugging. 
During  execution,  other  processes’  status  can  by  monitored  by  SYS,  which  then  reports  to  the  operator  on 
exceptional  situations  such  as  the  death  of  a process.  Currently,  the  SYS  initialization  code  is  included  in 
one  of  the  other  processes,  and  the  monitoring  functions  are  not  performed. 

3.5.  Debugging  Tools 

The  debugging  tools  are  provided  to  assist  in  program  development.  These  provide  user-friendly 
displays  of  the  state  of  the  common  memory  buffers  and  related  system  flags,  allow  variables  to  be  exam- 
ined and/or  changed  by  name,  obtain  a stack  trace  (listing  the  sequence  of  nested  subroutine  calls),  remove 
or  restore  named  sutwoudnes  or  execute  them  in  isolation,  and  set  debug  flags  which  release  additional 
output  during  program  execution. 

4.  Protocol  for  Interprocessor  Communications 

Central  to  the  CRAMPS  operating  system  are  the  inter-processw  communications,  which  are  imple- 
mented as  follows. 

Data  are  passed  from  one  processor  on  the  bus  to  anoth^  via  common  memory  (memory  accessible 
to  both  pocessors).  The  data  are  first  written  into  common  memory  by  the  one  processor,  and  then  read 
by  the  other.  It  is  the  principal  task  of  the  operating  system’s  communications  programs  to  assure  first 
that  the  data  is  not  read  until  the  writCT  is  finished  writing,  and  then  that  the  memory  is  not  written  to  (by 
some  other  processor)  until  the  reading  is  complete. 

To  achieve  this,  each  separate  data  buffa  in  common  memory  is  accessed  by  the  user  as  though  it 
were  a Unix  file,  which  must  be  explicitly  opened  and  closed.  (We  use  the  wwd  "file"  interchangeably 
with  "buffer",  to  emphasize  the  similarity  with  Unix  file  manipulation.)  There  is  one  important  rule  which 
the  system  enforces:  only  one  user  may  have  access  to  a particular  bi^er  at  any  one  time.  This  is  gen- 
erally a r^tsonable  requirement:  if  one  perscm  is  still  writing  a buffer,  the  other  shouldn’t  be  reading  it;  if 
he’s  not  still  writing,  he  should  relinquish  the  buffer  (close  it).  This  requirement  is  relaxed  in  a derivative 
protocol  described  below  in  secticm  4.5. 

Buffers  are  pomanently  assigned  areas  in  common  memory.  The  buffer  names  (filenames),  their 
common  memory  addresses,  legitimate  usos,  etc,  are  specified  in  an  allfiles  array  maintained  by  the  sys- 
tem, and  resident  in  the  system  buffer  pamfile.  Each  process,  at  initialization,  extracts  from  the  allfiles 
array  the  infnmation  conconing  those  buffers  for  which  it  is  a legitimate  uso*.  This  infcHmation  is  stored 
internally  in  the  pnxKess’s  own  on-board  files  array,  so  that  it  is  available  fcH*  use  during  real-time  process- 
ing without  the  added  bus  traffic  and  access  time  that  would  result  from  using  the  system  copy.  The  crea- 
tion of  the  files  array  is  done  by  the  system  at  the  beginning  of  each  process,  and  is  transparent  to  the 
user.  The  ^vantage  to  diis  centralized  scheme  is  that  changes  to  the  buffer  assignments  need  be  made 
only  in  allfiles,  and  user  programs  do  not  need  to  be  recompiled  afto'  such  changes. 

4.1.  Unix  Framework 

The  basic  subroutine  calls  (open,  close,  read,  write)  have  been  chosen  to  be  identical  to  the 
corresponding  ClUnix  (Syston  V)  system  calls.  This  is  a convenient  and  widely  known  quasi-standard. 
Many  {HOgrammers  will  be  faniliar  with  the  usage,  and  will  find  the  CRAMPS  procedures  easy  to 
remember.  Also,  this  chmce  allows  programs  to  be  compiled  and  tested  under  Unix  without  requiring 
changes  when  transporting  them  to  a target  board. 

These  basic  calls,  fiiUy  analogous  to  the  ccareqxmding  C calls,  are: 

file_descrq)tcH’  = qpen(filename,  mode); 
number_of_chars  = read(file_descriptor,  buffer,  count); 
number_of_chars  = HTfre(file_descriptOT,  buffer,  count); 
retum_code  = c/ose(file_descTiptor); 

Note  that  the  bi^er  argument  above  refers  to  the  address,  within  the  program,  that  one  reads  into  or  writes 
frxxn.  The  common  memory  buffo-  is  specified  by  the  filename  argument 
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The  openO  and  closeQ  calls  provide  the  mechanism  for  the  system  to  enforce  the  rule  that  at  most 
one  program  may  have  a given  buffer  open  at  any  one  time.  Thus  the  openQ  function  will  check  whether 
any  other  CPU  has  the  buffer  open.  If  so,  it  polls  until  the  buffer  is  closed.  Only  then  does  it  open  the 
buffer  and  return  a small  positive  number,  which  is  used  like  a Unix  file-descriptor,  to  the  calling  program. 

Since  openQ  waits  until  a buffer  is  free,  an  alternative  call,  opennQ  (open-or-retum-after-n-tries), 
provides  a way  to  try  to  open  a buffer  but  to  continue  with  other  processing  if  it  is  busy.  A return  of  -2 
from  openn  means  the  buffer  is  busy;  this  is  not  an  error  condition. 

Error  conditions  are  flagged  by  -1,  and  should  be  checked  for  as  in  Unix.  However,  error  messages 
(and  optionally  stack  traces)  are  printed  by  the  system,  so  the  user  need  not  add  additional  code  to  print 
messages.  Successful  readQ  and  writeQ  calls  return  the  number  of  bytes  transferred.  readQ  and  writeQ 
check  the  system’s  flag  area  to  be  sure  the  file  has  been  opened,  and  to  the  correct  user;  this  checking  can 
be  turned  off,  to  achieve  optimum  speed,  once  a program  has  been  sufficiently  debugged.  Similarly,  an 
open  JdQ  call  can  replace  the  openQ,  to  eliminate  the  search  through  the  files  array,  once  a buffer’s  loca- 
tion in  that  array  has  been  fixed. 

4.2.  Protocol  Implementation 

The  openQ  and  closeQ  functions  are  implemented  by  me^s  of  a one-byte  flag  associated  with  each 
buffer.  This  flag  is  zero  when  the  buffer  is  closed,  and  hex  80  (bit  7 set)  when  the  buffer  is  open.  When  a 
particular  user  attempts  to  open  the  buffer  (put  a 7 in  bit  7 of  the  flag),  he  does  so  by  means  of  a test- 
and-set  (JAS)  instruction  (Figure  1).  This  instruction,  on  the  680x0,  first  fetches  bit  7 of  the  specified 
address  (the  one-byte  flag),  and  then  sets  bit  7 (stores  a i in  it),  all  the  while  locking  other  users  out  of  the 
bus.  This  prevents  anyone  else  from  changing  the  flag  in  the  time  between  the  fetch  arxl  the  store  porticms 
of  the  instruction. 

Having  stored  a 7-bit  in  bit  7 of  the  currrat  flag,  the  user  then  determines  what  had  been  fetched 
from  that  flag  by  examining  the  negative-flag  in  the  680x0' s status  register.  If  it  was  a 7,  someone  else 
had  the  buffer  c^n;  this  user  has  not  changed  anything  (he  stored  a 7,  but  a 7 had  already  been  there).  He 
must  not  change  anything  in  the  buffer  or  its  associated  flags.  He  may  wait  and  try  again,  in  the  case  of 
openQ,  or  continue  processing,  in  the  case  of  opennQ. 

If  he  fetched  a 0,  on  the  other  hand,  this  means  the  buffer  was  closed.  It  is  now  open  to  this  usct, 
since  he  has  changed  bit  7 of  the  flag  to  7.  He  immediately  puts  his  "user  ID"  in  the  adjacent  byte,  called 
the  previousjiser  byte. 

Saving  the  us^  ID  of  the  last  user  to  open  the  buffo'  is  necessary  fOT  the  initialization  procedure 
(see  next  section).  In  addidon,  the  previousjiser  byte  has  proven  useful  to  applications  inx)grams  in 
ascertaining  whether  data  is  "old"  (last  opened  by  the  inquiring  process),  ex  "new"  (last  opened  by  the 
other  user).  They  are  also  invaluable  in  drugging,  fo  example  to  see  who  in  fact  last  used,  and  perhaps 
is  not  relinquishing,  a given  buffer.  To  further  aid  debugging,  the  system  maintains  the  time  of  last  open 
and  last  close,  fo  each  buffer.  Maintenance  of  these  time  entries  can  also  be  switched  off  once  they  are 
no  longer  needed. 

Note  that  the  flags  need  not  be  adjacent  to  the  buffers  themselves.  In  fact,  in  our  vision  application 
we  have  chosen  to  locate  these  flags  all  together,  so  that  the  status  of  all  of  the  system’s  buffers  may  be 
seen  at  a single  glance.  This  is  no  longer  important,  since  the  tools  now  include  a display  system  flags 
command.^ 

4J.  Execution  Hme  The  basic  protocol  is  very  fast  Several  subroutines  were  run  on  a 68020 
microprocessOT,  running  with  a 25  MHz  clock.  The  open  and  close  primidves  (including  maintaining  the 
"previous  user"  byte)  run  in  4.5  and  2.2  |isec,  respeedvely.  The  complete  debug  versions,  including  run- 
time checks  on  the  parameters  and  on  the  internal  files  array,  run  in  100  and  40  nsec,  respeedvely.  (These 
routines  are  written  in  C and  their  execudon  time  is  compiler-dependent) 


3 On  ceruin  hardware  confignratioai  the  TAS  instiuctioD  does  not  lock  the  on-boerd  bos,  but  c»ly  the  backplane  bus  linking  the 
boards.  In  such  cases,  individual  flag  buffers  must  be  located  so  that  they  are  ofif -board  to  those  processon  needing  to  set  the  flags. 
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These  timings  may  be  compared  with  values  quoted  in  a recent  article  [14]  on  a multi-tasking  robot 
operating  system.  That  system  required  913  and  353  psec,  respectively,  for  message  send  and  message 
receive  system  calls. 

4.4.  Buffer  Initialization 

Initializing  the  flags  at  startup  requires  care.  The  design  requirements  state  that  any  process  may  be 
started  up  independently  of  the  others.  That  is,  any  process  may  be  reset  and  restarted  while  the  others 
are  running;  and  processes  may  be  started  in  any  order.  Achieving  this  goal  required  that  each  buffer 
have  one  additional  parameter,  its  owner.  In  cases  where  it  is  not  otherwise  clear  which  process  is  respon- 
sible for  cleaning  up  a buffer’s  flags,  the  owner  is  designated  to  do  the  job. 

Specifically,  when  a process  is  first  started,  it  looks  at  each  of  its  buffers  (this  occurs  in  the  system 
prologue  that  is  automatically  linked  to  the  user’s  program).  If  the  previous_user  byte  of  a particular 
buffer  contains  that  user’s  own  ID,  it  initializes  the  buffer.  If  the  previous  user  was  another  user  assigned 
to  the  buffer,  the  job  is  left  to  that  otho-  user.  (That  other  user  may  have  just  written  good  data,  for 
example,  or  he  may  have  crashed  and  been  restarted—this  user  doesn’t  know,  only  the  other  user  knows.) 
Otherwise,  the  flags  are  junk  (such  as  the  FFFF  that  often  tq)pears  at  power  up).  If  this  process  is  the 
owner,  it  cleans  up  the  flags;  otherwise  it  does  nothing.  This  algorithm  identifies  one  and  only  one  pro- 
cess to  initialize  any  given  buffer  and  its  flags.  No  determination  of  "who  gets  there  first"  needs  to  be 
made. 

During  buffer  initialization,  there  is  one  case  in  which  a user  must  write  to  a flag  buffer  without 
having  first  formally  opened  the  buffer.  This  occurs  when  the  user  is  the  owner,  and  finds  the  buffer’s 
OPENICLOSED  flag  set  to  OPEN,  but  the  previousjiser  byte  is  garbage  (i.e.  not  a valid  user  of  that 
buffer).  He  goes  to  open  the  buffer.  The  problem  is,  it  may  be  that  the  OPEN  flag  is  validly  set  in  the 
course  of  anotho*  usct’s  open  of  the  buffer,  but  the  other  uso*  has  not  yet  put  his  own  userid  in  the 
previousjiser  byte.  The  solution  is  not  to  allow  any  user  to  open  a buffer  with  a garbage  previousjiser 
byte,  or  a garbage  flag  for  that  matter.  If  the  initialization  process  SYS  (see  Section  3)  is  run,  no  buffers 
are  left  uninitialized. 

An  even  more  insidious  case  can  occur  if  a process  wakes  up  to  find  a flag  set  to  OPEN,  with  the 
previousjiser  set  to  the  process’s  own  userid.  Hoe  again,  it  may  be  that  another  user  is  in  the  process  of 
opening  the  buffer,  and  has  set  the  flag  to  OPEN  but  has  not  yet  put  his  userid  in  the  previousjiser  byte 
(compare  this  with  the  much  mwe  likely  situation,  that  this  process  had  died  while  the  file  was  open  to 
him).  The  presence  of  this  user’s  userid,  althou^  this  user  is  just  now  initializing,  can  be  explained  as 
coming  from  a "in^vious  incarnation",  Le.  before  he  was  stopped  and  restarted.  Our  solution  to  this  ambi- 
guity, however  inelegant,  is  for  the  [vocess  just  starting  q)  to  go  to  sleep,  and  then  re-awaken  and  see  if 
the  flag  has  changed  or  not  If  not  he  then  presumes  it  to  be  indeed  open  to  him,  and  initializes  and  closes 
the  buffer. 

Buffers  are  zeroed  at  initialization  by  the  owner,  if  he’s  also  the  one  doing  the  initializing  of  the 
flags,  or  by  the  system  process  SYS  at  powerup.  The  various  inifializing  functions  are  called  by  a sulnou- 
tine  vmain,  which  is  automatically  included  by  the  linker  and  started  by  the  downloader/PROM.  This  ini- 
tialization is  tran^)arent  to  the  user. 

4.5.  Multiple*Reader  Buffers 

The  buffers  described  above  may  have  any  number  of  users  permitted  to  access  them.  In  the  proto- 
col just  described,  however,  only  (me  uso*  is  permitted  to  access  a given  file  at  any  one  time.  A user,  even 
if  he  is  (Mily  reading,  shuts  out  other  users,  even  those  who  ate  also  waiting  only  to  read 

Multiple-reader  bikers,  on  the  other  hand,  allow  several  readers  to  have  a file  open,  for  reading 
only,  at  the  same  time.  For  writing  into  the  file,  however,  a user  must  still  be  the  only  one  to  have  the  file 
open. 

The  fact  that  a buffer  pomits  multiple  readers  is  transparent  to  the  user.  He  still  uses  open  or 
openn  to  open  a buffer,  and  chse  to  close  it  His  int^nal  files  array  is  imchanged,  excq)t  that  now  the 
entry  multiple  is  set  to  TRUE. 
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The  multiple-reader  protocol  is  implemented  by  adding  a flag  called  rdusers,  which  lists  users  to 
whom  the  file  is  currently  open.  Each  bit  in  rdusers  is  assigned  to  a user,  and  is  set  if  that  user  has  the 
file  open.  To  prevent  contention  when  two  users  try  to  change  the  rdusers  flag  at  the  same  time,  the  pro- 
tocol described  in  the  previous  subsections  is  appplied  to  the  rdusers  flag  (actually  to  the  whole  flagbuf 
structure  containing  it),  instead  of  to  the  buffer.  Thus  rdusers  can  be  changed  by  only  one  user  at  a time. 
Note  that  previousjuser  now  refers  to  the  last  user  to  open  the  flagbuf  structure,  rather  than  the  last  user  to 
open  the  message  buffer. 

To  the  system,  the  open  flag  (and  also  the  previousjiser  byte)  are  taken  to  refer  to  just  the  flagbuf 
structure,  rather  than  the  whole  buffer.  That  is,  only  one  {H'ocess  may  write  into  or  read  from  the  flagbuf 
structure  at  a time,  namely  that  process  that  has  the  flagbuf  structure  open,  and  whose  name  therefore 
appears  in  the  previousjiser  byte.  The  same  open/close  protocols  are  us^,  but  they  refer  to  the  flagbuf 
structure  as  the  entity  being  opened  ot  closed. 

Once  the  flagbuf  structure  is  open  to  you,  you  may  modify  the  rdusers  vector  (the  open  call  does 
this  automatically)  to  add  cw  delete  yourself  as  a current  reader,  or  you  may  request  write  privileges.  If 
you  request  write  privileges,  you  will  be  kept  waiting  until  all  other  current  users  have  closM  the  buffer. 
No  new  users  will  be  admitted  until  you  close  the  buffer:  openn  will  return  FILEBUSY  status  (-2)  to  pros- 
pective new  users.  Again,  this  protocol  is  internal  to  the  system  functions  openQ  and  closeQ;  the  user 
need  not  even  know  whether  a given  buffer  is  a "multiple-reader  buffer"  w not 

The  openQ  and  closeQ  of  a multiple-reader  buffer  will  clearly  take  longo*  than  for  a regular  buffer, 
so  the  multiple-reader  facility  should  not  be  used  to  all  files.  The  advantage  of  this  protocol  accrues  when 
several  users  all  need  to  read  the  same  file,  and  it  is  not  updated  often.  An  obvious  candidate  is  parmfile, 
the  system  parameter  file.  This  is  written  by  SYS  during  initialization,  and  then  read  by  all  othw  processes 
as  needed. 

The  "multiple-reader"  capability  is  a switchable  option. 

4.6.  Dynamic  Common  Memory  Allocation 

The  use  of  dynamic  memory  allocation  to  intoprocessor  communication  is  quite  elegant  Suppose 
process  A has  data  that  he  wishes  to  send  to  processes  B,  C,  and  D.  He  requests  a block  of  common 
memory  firom  the  system,  in  the  amount  he  requires  to  his  data,  using  the  olloccmQ  call.  In  so  doing,  he 
also  informs  the  system  who  will  be  using  these  data,  namely  B,  C,  and  D.  The  system  returns  with  a 
pointer  to  a block  of  memoy  of  the  requested  size,  and  keeps  a list  of  the  prospective  users. 

At  this  point  A is  the  only  process  that  knows  about  (has  the  address  of)  this  block  of  memory.  He 
goes  ahead  and  writes  his  data  there;  he  does  not  have  to  open  it  like  a buffer,  since  no  other  process  even 
knows  of  its  existence.  When  he  is  done  writing,  he  passes  a pointer  to  the  block  to  the  appropriate 
processes,  B.  C,  and  D.  (He  also  notifies  the  system  that  he  is  done  with  the  block.) 

The  data  is  now  available  to  be  read.  Since  no  writers  have  access  to  the  block,  readers  need  not 
worry  that  the  data  will  change.  They  too  need  not  open  a close  the  data  buffer.  Moreover,  all  the 
readers  may  consider  the  block  to  be  open  to  them  simultaneously.  The  bus  hardware  arbitrates  each  fetch 
request  so  that  only  one  word  of  data  is  fetched  at  a time;  no  further  software  [votocol  is  required. 

Whenever  each  process  is  finished  using  the  data,  it  notifies  the  system  by  fireeing  the  block,  with  a 
call  to  freecmQ.  The  system  maintains  the  list  of  those  processes  using  the  block  of  memory  (initially  A 
through  D,  in  the  exam^e).  When  the  last  process  frees  it,  the  system  returns  the  block  to  the  pool  of  free 
memory,  from  which  it  can  be  reallocated. 

The  pointers  themselves  are  currendy  passed  in  buffers  of  the  kind  described  above  (requiring  opens 
and  c/ores— see  secttoi  4.2).  Because  of  the  specially  simple  nature  of  the  messages  (just  a 32-bit  address), 
the  special  calls  readheader  and  writeheader  are  available  to  opai,  read  (or  write),  and  close  with  a single 
call.  We  anticipate  modifying  this  protocol  further,  so  that  the  buffers  themselves  will  varush,  and  the 
four-byte  pointers  will  be  placed  diiwtly  in  the  flagbirf  structure.  Rnally,  with  the  advent  of  32-bit  wide 
bus  hardware,  the  opens  and  closes  can  be  eliminated,  since  the  writing  of  the  (32-bit)  pointer  will  be 
done  in  a single,  indivisible  bus  cycle;  the  reacto  will  always  read  the  latest  pointer,  and  will  never  get 
garbage  (i.e.  16  bits  of  new  pointer,  and  16  bits  of  old). 
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In  CRAMPS,  the  dynamic  common  memory  allocation  is  implemented  as  part  of  the  kernel,  which 
is  resident  on  each  process.  There  is  no  centralized  system  process  to  perform  such  allocation  for  the 
whole  bucket.  Instead,  each  process  is  given  a region  in  common  memory  from  which  to  carve  out  alloca- 
tion blocks  for  himself,  as  needed.  The  alternative,  of  having  the  575  process  handle  all  allocation 
requests,  has  not  been  pursued.  This  choice  is  transparent  to  the  user:  he  simply  calls  the  system  function 
alloccmO  and  receives  a pointer  to  memory  in  return. 

5.  Additional  Functions 

The  system  also  {S'ovides  I/O  services  to  and  firom  the  CRT,  including  getchar,  putchar,  printf,  and 
readnl.  In  each  case  it  first  checks  to  see  whether  a CRT  is  in  fact  hooked  up.  This  allows  the  user  to 
include  diagnostic  printf  statements  in  his  program,  while  still  being  able  to  run  in  real-time  (i.e.  without 
the  slow  terminal  HO)  simply  by  unplugging  the  terminal  from  the  computer  board. 

A system  clock  is  created  by  the  575  process,  and  continues  to  run  on  one  board,  interrupt  driven 
and  therefore  transparent  to  the  user.  This  clock  may  be  read  by  any  user  by  calling  a function  systime{)\ 
as  currently  implemented,  it  ticks  once  each  millisecond.  A sleep(n)  function  puts  the  process  to  sleep  for 
approximately  n milliseconds.  abort( message)  and  exit(n)  perform  as  expected. 

A PROC  (per  jHwess)  array,  maintained  in  common  memOTy,  irovides  a snapshot  of  the  current 
state  of  each  process.  A tools  command  displays  this  information  on  request 

Every  error  detected  by  the  system  gives  rise  to  a message  to  the  CRT  (if  there  is  one)  including  the 
name  of  the  subroutine  and  the  error,  and  relevant  parameters.  If  the  debug  flag  is  set  a stack  trace  is  also 
automatically  printed.  Thus,  while  a user  must  ch^k  for  a -1  oror  return,  he  need  not  bother  coding  an 
error  message.  (The  system  error  messages  may  be  disabled,  if  desired,  by  clearing  a flag.) 

The  same  buffer  protocol  as  described  above  has  also  been  implemented  for  host-target  data 
transfers,  to  allow  programs  to  be  debugged  on  the  host  or  to  pomit  the  host  to  influence  execution  on 
the  target  Since  the  host  caimot  directly  lock  the  bus,  the  CPU  on  the  interface  board  (ruiming  code  in 
its  PROM)  accepts  a request  from  the  host  to  do  the  test-and-set  on  the  host’s  behalf,  and  to  return  the 
result  to  the  host 

Additional  buffer  ("file")  functions  are  provided,  such  as  readran  and  writeran,  which  provide  ran- 
dom access,  and  autoread  and  autowrite,  which  include  the  open  and  close  functions  automatically,  and  so 
result  in  fewer  programmer  errors.  Open_syhch,  togeths  with  the  global  variables  synchrjxise  and 
synchrjncr,  allows  the  user  to  require  that  message  passing  be  done  at  fixed  times  only  (synchronously). 

These  and  other  functions  are  described,  and  directions  for  their  use  is  given,  in  The  CRAMPS 
Operating  System:  User’s  Guide  [12].  Details  of  how  to  set  up  the  files  arrays,  uso-  IDs,  etc,  will  be 
presented  in  The  CRAMPS  Operating  System:  Administrator’s  Guide  [13]. 

6.  Application 

CRAMPS  - an  earlier,  less  developed  version  - was  used  as  the  (grating  system  in  the  vision  sys- 
tem component  of  the  National  Bureau  of  Standards  Automated  Manufacturing  Research  Facility  (AMRF). 
The  AMRF  is  a fully-automated  machine-tool  shop  designed  as  a demonstration  and  test-bed  facility.  The 
vision  system  was  built  and  tested  in  1982-1985,  and  continues  in  operation  today.  Hardware  consists  of 
five  CPU  boards  nmning  Intel  8086  inicrq)rocess(ws  in  a Multibus  backplane,  and  a separate  host  com- 
puter consisting  an  ^)86  CPU  board  in  an  S-100  backplane,  communicating  with  the  Multibus  bucket  in 
real-time  as  discussed  in  the  |vevious  section. 

Program  executkm  in  the  vision  system  was  data-driven.  Each  processor  performed  a different  func- 
tion on  the  image.  One  did  run-length  encoding  of  the  image  (see  Ref.  [3]),  die  next  performed  segmenta- 
tion, the  third  extracted  features  such  as  comers,  etc.  Images  arrived  from  the  camera  30  times  each 
second.  Each  image  was  inocessed  by  the  first  processtx’.  The  output  of  this  stage  was  then  passed  (via  the 
CRAMPS  cperadng  system)  to  the  next,  etc.  The  receiving  {nxx^essOT  is  always  available  (at  its 
programmer’s  discretion)  to  deal  with  the  incoming  data  There  is  never  any  conc^  that  it  might  be 
swiped  out  of  memory,  or  when  it  will  again  gain  control  of  the  processor,  or  how  to  presme  the  data 
until  it  is  again  running.  Intmupts  were  not  used;  each  process  simply  checked  to  see  if  new  data  was 
available,  at  appropriate  points  in  its  code.  The  opoating  system’s  responsibility  is  simply  to  provide 
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bookkeeping  for  the  data  passing,  and  to  assure  that  there  are  no  collisions.  CRAMPS  does  this  with  low 
overhead,  and  what  overhead  there  is  occurs  at  points  explicitly  recognized  by  the  user,  i.e.  during  a call 
to  read  or  write. 

Since  then  CRAMPS  has  been  converted  to  run  on  680x0  processors,  and  its  capabilities  have  been 
greatly  enhanced.  Three  processes  of  the  AMRF  vision  system  were  transported  to  three  68010's,  where 
they  ran  under  CRAMPS  as  before.  Other  programs  were  written  to  test  some  of  the  new  capabilities  of 
CRAMPS,  and  they  demonstrated  the  usefulness  of  these  new  features.  CRAMPS  has  thus  proved  itself  to 
be  dependable,  as  well  as  fast,  on  both  8086's  and  680x0's. 
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Fig.  1 . Diagram  of  TAS  (test  and  set)  instruction. 
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